Let M denote M = 2 1 1 2 1 . . . . . . . . . 1 2 1 1 2 R

Modeling dependencies in multivariate discrete data is a challenging problem, especiallyinhighdimensions.

Contrary to common belief, we show that gradient ascent-based unconstrained optimization methods frequently fail to perform machine unlearning, a phenomenon we attribute to the inherent statistical dependence between the forget and retain data sets. This dependence, which can manifest itself even as simple correlations, undermines the misconception that these sets can be independently manipulated during unlearning. We provide empirical and theoretical evidence showing these methods often fail precisely due to this overlooked relationship. For random forget sets, this dependence means that degrading forget set metrics (which, for a retrained model, should mirror test set metrics) inevitably harms overall test performance. Going beyond random sets, we consider logistic regression as an instructive example where a critical failure mode emerges: inter-set dependence causes gradient descent-ascent iterations to progressively diverge from the ideal retrained model. Strikingly, these methods can converge to solutions that are not only far from the retrained ideal but are potentially even further from it than the original model itself, rendering the unlearning process actively detrimental. A toy example further illustrates how this dependence can trap models in inferior local minima, inescapable via finetuning. Our findings highlight that the presence of such statistical dependencies, even when manifest only as correlations, can be sufficient for ascent-based unlearning to fail. Our theoretical insights are corroborated by experiments on complex neural networks, demonstrating that these methods do not perform as expected in practice due to this unaddressed statistical interplay.

A.2 Proof of Relation (3) We can write D One class of transport maps we consider in our numerical experiments (i.e., to approximate Another underlying class of transports that we use in our numerical experiments are inverse auto-regressive flows (IAFs). IAFs are built as a composition of component-wise affine transformations, where the shift and scaling functions of each component only depend on earlier indexed variables. Flows are typically comprised of several IAF stages with the components either randomly permuted or, as we choose, reversed in between each stage. Here we discuss how generalized linear models may naturally admit lazy structure. Here we describe the numerical algorithms required by the lazy map framework.

A successful approach to structured learning is to write the learning objective as a joint function of linear parameters and inference messages, and iterate between updates to each. This paper observes that if the inference problem is "smoothed" through the addition of entropy terms, for fixed messages, the learning objective reduces to a traditional (non-structured) logistic regression problem with respect to parameters. In these logistic regression problems, each training example has a bias term determined by the current set of messages. Based on this insight, the structured energy function can be extended from linear factors to any function class where an "oracle" exists to minimize a logistic loss.